Development of a situational judgment test to supplement current US air force measures of officership.

Aptitude requirements for US Air Force officer commissioning include completion of a college degree and minimum scores on the Air Force Officer Qualifying Test (AFOQT) Verbal and Quantitative composites. Although the AFOQT has demonstrated predictive validity for officer training, the Air Force has striven to improve predictive validity and diversity. To this end, a Situational judgment Test (SJT) was added to the AFOQT in 2015. SJT development was consistent with recommendations to broaden the competencies assessed by the AFOQT with the goal of providing incremental validity, while reducing adverse impact for historically underrepresented groups. To ensure content validity and realism, SJT development was based on competencies identified in a large-scale analysis of officership and input from junior officers in scenario and response generation and scoring. Psychometric evaluations have affirmed its potential benefits for inclusion on the AFOQT. An initial study showed the SJT to be perceived as highly face valid regardless of whether it was presented as a paper-and-pencil test (with narrative or scripted scenarios) or in a video-based format. Preliminary studies demonstrated criterion-related validity within small USAF samples, and a larger Army cadet sample. Additionally, operational administration of the SJT since 2015 has demonstrated its potential for improving diversity (i.e., reduced adverse impact relative to the AFOQT Verbal and Quantitative composites). Predictive validation studies with larger Air Force officer accession samples are ongoing to assess the incremental validity of the SJT beyond current AFOQT composites for predicting important outcomes across accession sources.

Validation and refinement of the "Airman Comprehensive Assessment": Evaluating competency proficiencies of enlisted members.

Beyond proficiency on occupationally specific tasks, the U.S. Air Force expects members to develop proficiency on institutionally valued "soft skill" competencies (e.g., Teamwork, Communication, and Initiative) throughout their careers. As such, all E1-E6 members are annually evaluated using Behaviorally Anchored Rating Scales (BARS) designed to measure such competencies. Despite mandated use, these Airman Comprehensive Assessment (ACA) scales previously have not been empirically evaluated. To address this gap, we surveyed Air Force supervisors, using a criterion-related sampling methodology to validate the behavioral anchors for each scale. Supervisors identified two subordinates of the same rank/career field who they viewed as having (a) high potential for future success in an Air Force career or, alternately, (b) lower potential for future career success and rated each subordinate on the individual behaviors that comprise the 12 scales. ACA items were intermixed with scale items previously identified as distinguishing top performers in civilian organizations. Results demonstrate scale reliability and generally validate the ACA competency scales as stronger differentiators of supervisor-rated career potential than competency scales developed for civilian organizations. We provide recommendations for re-calibration of scale anchors based on the relative percentage of high vs. low potential members that demonstrate each behavior, and suggest changes to improve correspondence between measured competency proficiency and supervisor-rated career potential.

Defining occupation-specific performance components for military selection and classification.

Differentiating performance components across military occupations is critical to developing assessments and other applications for matching people to occupations in the military. However, identifying occupation-specific performance components is challenging and resource intensive. The current paper summarizes different methods the services use to define and identify occupation-specific performance components. The paper concludes with recommendations and future directions for advancing the military's needs for information and data on occupation-specific performance components.

Improving the effectiveness of embedded behavioral health personnel through situational judgment training.

Embedding behavioral health (EBH) personnel into operational units has emerged as a major trend within the US military. These positions require skillsets in addition to those needed in a clinic setting. Little or no empirically based training has yet been developed to ensure preparedness to serve in EBH roles. Based on the demonstrated effectiveness of situational judgment tests (SJT) in training for other domains, we developed and evaluated 60 SJT scenarios for use in the US Air Force's formal EBH training course. SJTs were based on real-world experiences of seasoned EBH professionals, and were used for structured, scenario-based discussions during training as well as a 30-item pre- and posttest to assess learning. Use during training was associated with significantly improved scores in six EBH competencies (ethical decision-making, leadership consultation, balancing professional relationships, unit engagement, triage to correct services, and teaming). In a follow-up survey, students reported scenario-based discussions were very helpful, and reported improved abilities and confidence. This study demonstrates the potential utility of the SJT methodology to prepare mental health professionals to serve in EBH positions. Further development and possible applications are discussed, such as expansion to other training venues, job candidate screening, and assessment for readiness for EBH practice.

Multitasking as a predictor of simulated unmanned aircraft mission performance: Incremental validity beyond cognitive ability.

Simulations are increasingly popular in employee selection and training. While face valid and engaging, the attributes being assessed are often poorly understood. This study evaluated the extent to which a multitasking assessment based on concurrent memorization, math, visual monitoring, and listening tasks predicted simulated unmanned aircraft vehicle (UAV) mission performance in a military trainee sample (N = 368). Performance was based on accuracy of mission planning, information recall during "Lost Link" conditions, and success in rescuing stranded allies while monitoring the aircraft's resources. Although scores on the multitasking assessment were only weakly related to performance of pre-flight mission planning tasks completed under static conditions, multitasking was strongly related to overall simulated UAV mission performance, including execution of tasks requiring attending to multiple, dynamic sources of information and shifting attention among concurrent processes and demands. Further, multi-tasking demonstrated substantial incremental validity beyond the traditional measures of cognitive ability that have been used for decades within the US military. Implications, limitations, and recommendations for selection and classification and future research are discussed.

Tailored Adaptive Personality Assessment System (TAPAS) as an indicator for counterproductive work behavior: Comparing validity in applicant, honest, and directed faking conditions.

The predictive validity of the Tailored Adaptive Personality Assessment System (TAPAS), the U.S. Army's first computer-adaptive personality test incorporating multidimensional pairwise preference items, has been demonstrated for training performance in both the Army and Air Force. While the unique TAPAS format has been described as more resistant to applicant faking than traditional self-report personality measures, evidence regarding the magnitude of applicant score distortion on TAPAS, and how such distortion (if present) may affect reliability and validity, has been limited. To address this gap, the present study compared operational TAPAS scores of Air Force enlisted recruits (administered pre-accession to applicants) to their post-accession retest scores under honest and directed faking ("fake good") conditions (based on re-administration of TAPAS during Basic Military Training). Data are presented on the relationship of applicant pre-accession scores to their retest scores under honest conditions (a form of test-retest reliability) and the magnitude of mean score differences in applicant, honest, and directed faking conditions is documented. Further, the validity of the TAPAS as an indicator for counterproductive work behaviors (CWB) was evaluated. Results indicate that TAPAS scores are relatively stable over time and the TAPAS methodology appears to reduce score distortion. In addition, the results suggest that the validities of the TAPAS scores as CWB correlates are comparable across honest and directed faking testing conditions and generally in line with those found for traditional Likert-type self-report Big Five measures.

Air Force Work Interest Navigator (AF-WIN) to improve person-job match: Development, validation, and initial implementation.

This article describes development and validation of a web-based vocational interest tool designed to help recruits and re-trainees identify enlisted career fields that match their preferences for work contexts, activities, and functional roles in support of the Air Force mission. The tool has recently been implemented for use by members considering re-training, and is undergoing pilot testing for potential use in the recruiting process. We first describe how the AF-WIN was developed, based on adaptation of the taxonomy from a Navy vocational interest tool (Navy's Job Opportunities in the Navy [JOIN]), followed by surveys of subject matter experts (SMEs) in 132 Air Force career fields on relevant job markers. We then describe a validation study in which job incumbents completed the AF-WIN and reported their level of job satisfaction within their current career field; results show that incumbents the AF-WIN algorithm identified as a good match for their career field reported substantially higher levels of job satisfaction than incumbents identified as a relatively poor match based on the tool. Finally, we provide results from initial beta-testing of the tool in a sample of recent enlisted trainees on perceived accuracy, utility, and functionality of the tool for use in the initial job assignment process.

Effects of manifest ethnic identification on employment discrimination.

Evidence from recent laboratory experiments suggests that ethnic identification can lead to negative evaluations of ethnic minorities (Kaiser & Pratt-Hyatt, 2009). The current research considers the generalizability of these findings to face-to-face interactions in contexts wherein impression management concerns are salient: the workplace hiring process. In a field experiment, Black, Hispanic, and Irish individuals applied for retail jobs with or without visible display of their ethnic identification. Analysis of indicators of formal (e.g., application offering, interview scheduling) and interpersonal discrimination (e.g., interaction length, nonverbal negativity) suggest store personnel interacting with other-race applicants exhibited greater positivity and longer interactions when applicants displayed ethnic identification than when they did not. The findings suggest that psychologists need to understand not only attitudes or intentions expressed in the lab, but also the behavioral consequences of manifest group identity as they unfold in natural environments.

Making those who cannot see look best: effects of visual resume formatting on ratings of job applicants with blindness.

OBJECTIVE: Although general attitudes toward individuals with disabilities are often positive, these perceptions do not always lead to equal footing in the hiring process. This study examined stereotypes of job applicants perceived to be blind and the role of applicant blindness in hireability ratings made by human resource managers. Specifically, we highlighted a unique challenge for individuals who cannot see: the visual formatting of resumes. DESIGN: Human resource managers (N = 249) evaluated the visually formatted or unformatted resumes of hypothetical job applicants who were portrayed as blind or sighted and rated applicant hireability and personality characteristics. RESULTS: Although applicants perceived to be blind were perceived as more conscientious and agreeable by human resource managers, these positive evaluations did not translate into favorable hireability evaluations. CONCLUSION: Because human resource managers severely penalize applicants who do not attend to visual, nonfunctional resume presentation, applicants who cannot see are apt to find themselves disadvantaged in the hiring process. The implications of these findings for organizations, job seekers, and rehabilitation professionals are discussed.